The Increasing Risks of Risk Assessment: On the Rise of Artificial Intelligence and Non- Determinism in Safety-Critical Systems

Risk assessment plays a key role in Safety Management Systems. For more than forty years, likelihood and consequence have been used to guide the allocation of finite resources. Standards, such as IEC61508 and the DO-178 series, extended these concepts to support the development of software related systems. Human reliability analysis developed risk assessment techniques to represent and reason about operator ‘error’ and management failure. However, new challenges raise questions about the utility of traditional approaches to the development of safety-critical systems. The introduction of artificial intelligence within autonomous systems makes it hard to reason about the probability and consequences of adverse events when control applications must use previous training sets to guide their response to novel situations. This paper struggles to retain the foundations of risk assessment as a tool for safety engineering in the face of these new challenges for the development of safety-critical applications.